Autonomous vehicle user interface with predicted trajectories

ABSTRACT

Systems and methods are provided for generating trajectories for a vehicle user interface showing a driver&#39;s perspective view. Methods include generating an ego-vehicle predicted trajectory for an ego-vehicle; and generating at least one road agent predicted trajectory for a road agent that is external to the ego-vehicle. After the predicted trajectories are generated, the method continues by determining that at least one predicted trajectory overlaps either an object or another predicted trajectory, when displayed on the user interface showing a driver&#39;s perspective view. The method includes modifying the at least one road agent predicted trajectory to remove the overlap. The method then proceeds with updating a display of the user interface to include any modified road agent predicted trajectory. Systems include a trajectory-prediction module to execute the methods.

TECHNICAL FIELD

The present disclosure generally relates to user interfaces forautonomous vehicles and, more particularly, to systems and methods formodifying and displaying predicted trajectories on user interfaces thatprovide a driver's perspective view.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it may be described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presenttechnology.

In various applications, vehicular systems may predict the trajectory ofa vehicle (sometimes referred to herein as the “ego-vehicle”). Forexample, a parallel-autonomy vehicle, such as a vehicle that includes anadvanced driver-assistance system (ADAS), is a vehicle whose control maybe shared between a human driver and an autonomous-driving system. Thehuman driver may retain control of certain aspects of driving such avehicle (e.g., steering) while the ADAS monitors the driver's actionsand, when necessary, intervenes to prevent an accident. Predicting thetrajectory of the ego-vehicle is thus an important aspect of such anADAS. The vehicle system may display the ego-vehicle trajectories on auser interface.

The vehicular system may also predict the trajectory of one or more roadagent(s) external to a vehicle, and display the road agent trajectorieson the user interface display. Examples of road agents include varioustypes of other vehicles (e.g., automobiles, motorcycles, or bicycles)and pedestrians. One objective for an autonomous vehicle or aparallel-autonomy vehicle is to travel a route without colliding withthe road agents the vehicle encounters along the way. Since theintentions of road agents or their drivers are not usually known withcertainty to an autonomous vehicle or the driver of a parallel-autonomyvehicle, predicting the trajectory of a road agent can further thatobjective.

However, when many road agent trajectories are provided on a userinterface, it may quickly become overly complicated, especially when thedisplay is provided as a driver's perspective view (as compared to a topplan view, for example). Accordingly, it would be desirable to provideimproved trajectory prediction systems to adequately account for thepresence of numerous and/or overlapping trajectories that may beprovided on a user interface, which may lead to less complicateddisplays.

SUMMARY

This section provides a general summary of the disclosure, and is not acomprehensive disclosure of its full scope or all of its features.

In various aspects, the present teachings provide a system forgenerating trajectories for a vehicle user interface showing a driver'sperspective view. The system includes one or more processors, and amemory communicably coupled to the one or more processors. The memorystores a trajectory-prediction module including instructions that whenexecuted by the one or more processors cause the one or more processorsto perform a series of steps. For example, the trajectory-predictionmodule may include an instruction to generate an ego-vehicle predictedtrajectory for an ego-vehicle, and to generate at least one road agentpredicted trajectory for a road agent that is external to theego-vehicle. The instructions may include a step to determine that atleast one road agent predicted trajectory protrudes from a display areaand has an unknown direction of travel when displayed on the userinterface showing a driver's perspective view. The instructions mayinclude a step modify the at least one road agent predicted trajectoryto provide an indication of direction. A control module may also beprovided, including instructions that, when executed by the one or moreprocessors, cause the one or more processors to update the userinterface to include any modified road agent predicted trajectory.

In other aspects, the present teachings provide a system for generatingtrajectories for a vehicle user interface showing a driver's perspectiveview. The system includes one or more processors, and a memorycommunicably coupled to the one or more processors. The memory stores atrajectory-prediction module including instructions that when executedby the one or more processors cause the one or more processors toperform a series of steps. For example, the trajectory-prediction modulemay include an instruction to generate an ego-vehicle predictedtrajectory for an ego-vehicle, and to generate at least one road agentpredicted trajectory for a road agent that is external to theego-vehicle. The instructions may include a step to determine that atleast one predicted trajectory overlaps either an object or anotherpredicted trajectory, when displayed on the user interface showing adriver's perspective view. The instructions may include a step to modifythe at least one road agent predicted trajectory to remove the overlap.A control module may also be provided, including instructions that, whenexecuted by the one or more processors, cause the one or more processorsto update the user interface to include any modified road agentpredicted trajectory.

In still other aspects, the present teachings provide a method forgenerating trajectories for a vehicle user interface showing a driver'sperspective view. The method includes generating an ego-vehiclepredicted trajectory for an ego-vehicle; and generating at least oneroad agent predicted trajectory for a road agent that is external to theego-vehicle. After the predicted trajectories are generated, the methodcontinues by determining that at least one predicted trajectory overlapseither an object or another predicted trajectory, when displayed on theuser interface showing a driver's perspective view. The method includesmodifying the at least one road agent predicted trajectory to remove theoverlap. The method then proceeds with updating a display of the userinterface to include any modified road agent predicted trajectory.

Further areas of applicability and various methods of enhancing theabove technology will become apparent from the description providedherein. The description and specific examples in this summary areintended for purposes of illustration only and are not intended to limitthe scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present teachings will become more fully understood from thedetailed description and the accompanying drawings, wherein:

FIG. 1 is a schematic diagram illustrating an exemplary aspect of avehicle within which systems and methods disclosed herein according tothe present technology may be implemented;

FIG. 2 is a schematic diagram illustrating an exemplary aspect of atrajectory prediction system as provided in FIG. 1;

FIG. 3 illustrates a partial perspective view of an interior cabin of anexemplary vehicle interior compartment, providing multiple displaysystems that can be used, either singly or in combination, to provide auser interface display(s) according to various aspects of the presenttechnology;

FIGS. 4A-4E are five example displays illustrating a user interfacehaving an image representing a top plan view of an ego-vehicle and atleast one road agent vehicle or pedestrian with respective trajectories;

FIGS. 5A-5E are five example displays illustrating a user interfacehaving an image representing a front perspective view, or driver's view,of the ego-vehicle and the road agent vehicle(s) or pedestrians andtheir respective trajectories in the same scenarios as provided in FIGS.4A-4E, showing an overlap in trajectories and vehicles;

FIG. 6 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap of the predicted trajectorywith an object, in accordance with an illustrative aspect of the presenttechnology;

FIG. 7 is another flow chart diagram of a method of modifying at leastone predicted trajectory based on an overlap of the predicted trajectorywith an object, in accordance with an illustrative aspect of the presenttechnology;

FIGS. 8A-8C illustrate a modification of at least one predictedtrajectory based on an overlap with an obj ect, in accordance with anillustrative aspect of the present technology;

FIGS. 9A-9C illustrate a modification of at least one predictedtrajectory based on an overlap with two objects, in accordance with anillustrative aspect of the present technology;

FIG. 10 is flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap with an object by hiding,diluting, and/or diluting the predicted trajectory, in accordance withvarious aspects of the present technology;

FIGS. 11A-11E illustrate various modifications of at least one predictedtrajectory, in accordance with the methods of FIG. 10;

FIG. 12 is a flow chart diagram of a method of selecting differenttechniques of modifying at least one predicted trajectory based ondifferent overlaps, in accordance with various aspects of the presenttechnology;

FIGS. 13A-13B illustrate modifications of at least one predictedtrajectory, in accordance with the methods of FIG. 12;

FIG. 14 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap of at least two projectedtrajectories, with an optional use of confidence scores, in accordancewith an illustrative aspect of the present technology;

FIG. 15 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on a shorted distance between two adjacentpredicted trajectories, in accordance with an illustrative aspect of thepresent technology;

FIGS. 16A-16C illustrate modifications of at least one predictedtrajectory, in accordance with the methods of FIG. 14;

FIG. 17 illustrates a user interface display from a driver's perspectiveview with an overlap in road agents and predicted trajectories;

FIGS. 18A-18E illustrate modifications of at least one predictedtrajectory of FIG. 17, in accordance with the methods of FIGS. 14-15;

FIG. 19 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap of at least two projectedtrajectories, with a determination of a priority intersection, inaccordance with an illustrative aspect of the present technology;

FIG. 20 illustrates a set of four vehicles having predicted trajectorieswith four intersections having various priorities;

FIGS. 21A-21B illustrate modifications of at least one predictedtrajectory, in accordance with the methods of FIG. 19;

FIG. 22 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap of a projected trajectory andan object, with a determination that the object is a partially or fullyhidden road agent, in accordance with an illustrative aspect of thepresent technology;

FIGS. 23A-23D illustrate a modification of at least one predictedtrajectory, in accordance with the methods of FIG. 22;

FIGS. 24A-24D illustrate additional modifications of at least onepredicted trajectory, in accordance with the methods of FIG. 22;

FIG. 25 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on the presence of a sloped terrain, inaccordance with an illustrative aspect of the present technology;

FIGS. 26A-26D and 27A-27D illustrate modifications of at least onepredicted trajectory, in accordance with the methods of FIG. 25;

FIG. 28 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on at least one trajectory protruding from adisplay area, with an unknown direction of travel, in accordance with anillustrative aspect of the present technology;

FIGS. 29A-29E illustrate a modification of at least one predictedtrajectory, in accordance with the methods of FIG. 28;

FIG. 30 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap of a projected trajectory andeither an object or another predicted trajectory, with a determinationthat the predicted trajectory overlaps a plurality of static objects, inaccordance with an illustrative aspect of the present technology;

FIGS. 31A-31C illustrate a modification of at least one predictedtrajectory, in accordance with the methods of FIG. 30;

FIG. 32 is a flow chart diagram of a method of modifying at least onepredicted trajectory based on an overlap of a projected trajectory andeither an object or another predicted trajectory, with a determinationthat the predicted trajectory is a past tense road agent trajectory, inaccordance with an illustrative aspect of the present technology;

FIGS. 33A-33C illustrate a modification of at least one predictedtrajectory, in accordance with the methods of FIG. 32;

FIG. 34 is a flow chart diagram of a method for selecting a type ofdisplay to be provided in a user interface based on complexity, inaccordance with an illustrative aspect of the present technology;

FIGS. 35A-35C illustrate variations of predicted trajectories havingdifferent lines, 2-D patterns, and 3-D shapes;

FIGS. 36A-36C illustrate 2-D and 3D predicted trajectories with anoverlap of two predicted traj ectories;

FIGS. 37A-37C illustrate 2-D and 3D predicted trajectories with anoverlap of a predicted trajectory and at least one static object; and

FIGS. 38A-38B illustrate the use of a combination of 2-D and 3Dpredicted trajectories on a single display.

It should be noted that the figures set forth herein are intended toexemplify the general characteristics of the methods, algorithms, anddevices among those of the present technology, for the purpose of thedescription of certain aspects. These figures may not precisely reflectthe characteristics of any given aspect, and are not necessarilyintended to define or limit specific aspects within the scope of thistechnology. Further, certain aspects may incorporate features from acombination of figures.

DETAILED DESCRIPTION

The technology described herein pertains to an improved display ofpredicted road agent and ego-vehicle trajectories on a user interface.In particular, the technology improves how those trajectories caninteract with one another and be adapted and/or simplified with respectto size, location, and various other details related to their display onthe user interface. In this regard, this technology can simplifytrajectory information presented to a user. For example, it can provideimproved trajectory prediction systems providing displays thatadequately account for the presence of numerous and/or overlappingtrajectories that may be provided on a user interface, and may lead toless complicated views.

As used herein, the term “trajectory” or “trajectories” can refer topast, current, and future trajectories simulated, predicted, or observedfor a given road agent, vehicle, ego-vehicle, etc. As used herein, theterm “road agent” refers generally to any object that is capable ofmoving from place to place along, or in a manner that intersects with, aroadway. Such objects are not always necessarily in motion. For example,various aspects described herein consider an automobile, bus, bicycle,other type of vehicle parked along a street to be a road agent. In thoseaspects, the systems may track the parked automobile, along with otherdetected objects in the environment, using the vehicle's sensors. Thesensor data will typically reveal that the road agent (the parkedautomobile) is stationary—that there is no trajectory associated with itthat can be predicted at that time. However, in those various aspects,the system might continue to track the parked automobile because itcould begin moving at any time. In various aspects, the road agents ofinterest are external to a vehicle (sometimes referred to herein as the“ego-vehicle” or “host vehicle”) in which an aspect of the presenttechnology is operating. Such road agents are sometimes referred toherein as “external road agents.” Additional non-limiting examples ofroad agents include, without limitation, other vehicles of various types(automobiles, buses, motorcycles, bicycles, trucks, constructionequipment, etc.), pedestrians, and animals. In some aspects, a roadagent may simply be referred to as an object.

In non-limiting aspects, probabilistic variational trajectory predictorsmay be used to predict the ego-vehicle and/or road-agent trajectories,which may be referred to as predicted trajectories. In those aspects,the trajectory probability distributions for the ego-vehicle or a givenroad-agent, whichever applies, can be sampled to generate one or morespecific predicted trajectories. Those predicted trajectories can becross-fed and iteratively updated between the ego-vehicle and one ormore road agents, as described above, and they can also be output to acontrol module of the ego-vehicle that controls, to at least someextent, the output provided to various displays and user interfaces ofthe ego-vehicle, as described further below. In somevariational-predictor aspects, the statistical parameters of thetrajectory probability distributions may be output to the control moduleof the ego-vehicle instead of specific trajectories sampled from thedistributions.

Depending on the particular aspect, the ego-vehicle predictedtrajectories can be created considering the predicted trajectories ofmultiple external road agents in any of a number of possible orderings.In one aspect, the road-agent predicted trajectories are prioritized bytheir distance from the ego-vehicle, those closer to the ego-vehiclereceiving a higher priority than those farther away. In another aspect,the road-agent predicted trajectories are prioritized by any uncertaintyassociated with the road-agent predicted trajectories, those with lessuncertainty (i.e., greater certainty) receiving a higher priority thanthose with greater uncertainty (i.e., lower certainty). Furthermore,intermediate predicted trajectories for the ego-vehicle and/or one ormore external road agents during the iterative trajectory predictionprocess can be preserved, collected, and aggregated, taking into accountall possible orderings of the external road agents. Keeping all of thesevarious hypotheses alive permits the ego-vehicle's control module toconsider all of the possible actions the road agents might take. Thisconservative approach furthers the objective of the ego-vehicle planningand traversing a safe trajectory.

Other techniques can be combined advantageously with theiterative-trajectory-prediction architecture described above: (1)employing multiple trajectory predictors to predict the futuretrajectory of the ego-vehicle and multiple trajectory predictors topredict the future trajectory of one or more road agents external to theego-vehicle; and (2) generating confidence estimates for the predictedego-vehicle and road-agent trajectories so that their trustworthinesscan be evaluated. These techniques are explained further in theparagraphs that follow.

To predict the future trajectory of the ego-vehicle or a given externalroad agent, some aspects described herein employ two or more trajectorypredictors that use different deterministic or probabilisticcomputational models. For example, in one aspect including twotrajectory predictors, the first trajectory predictor is a probabilisticvariational trajectory predictor that includes a DNN, and the secondtrajectory predictor is a physics-based (deterministic) model. Invarious aspects, the trajectory predictors receive, as inputs, any of avariety of vehicle sensor data discussed further below. Depending on theparticular aspect, the trajectory predictors may also receive measuredpast trajectory information for the ego-vehicle or road agent, dependingon which type of trajectory is being predicted.

Regarding the confidence estimates, one important aspect of thedisclosed aspects is the temporal (time) horizon over which a vehicle orroad-agent trajectory is predicted. For example, a given predictedtrajectory from a particular trajectory predictor might be trustworthyover a relatively short temporal horizon of from about 0.1 to about 3seconds, but it might not be trustworthy over a longer temporal horizonextending beyond about 3 seconds up to about 10 seconds. In someaspects, the confidence estimates for the ego-vehicle and road-agentpredicted trajectories are computed as a continuous-time function overthe applicable temporal horizon using a deep-neural-network (DNN) model.The confidence measures thus assist the trajectory prediction system indeciding which ego-vehicle or road-agent predicted trajectories are mosttrustworthy for particular segments of the overall temporal predictionhorizon. In various aspects, the confidence scores associated with theiteratively updated ego-vehicle and road-agent predicted trajectoriesare also iteratively updated as the predicted trajectories themselvesare iteratively updated.

Referring to FIG. 1, an example of a vehicle 100 (sometimes referred toherein as an “ego-vehicle”) is illustrated. As used herein, a “vehicle”is any form of motorized transport. In one or more implementations, thevehicle 100 is an automobile. While arrangements will be describedherein with respect to automobiles, it will be understood that aspectsare not limited to automobiles. In some implementations, the vehicle 100may be any other form of motorized transport that, for example, canoperate at least semi-autonomously.

The vehicle 100 also includes various elements. It will be understoodthat in various aspects it may not be necessary for the vehicle 100 tohave all of the elements shown in FIG. 1. The vehicle 100 can have anycombination of the various elements shown in FIG. 1. Further, thevehicle 100 can have additional elements to those shown in FIG. 1. Insome arrangements, the vehicle 100 may be implemented without one ormore of the elements shown in FIG. 1. While the various elements areshown as being located within the vehicle 100 in FIG. 1, it will beunderstood that one or more of these elements can be located external tothe vehicle 100. Further, the elements shown may be physically separatedby large distances. Some of the possible elements of the vehicle 100 areshown in FIG. 1 and will be described along with subsequent figures.However, a description of many of the elements in FIG. 1 will beprovided after the discussion of the remaining figures for purposes ofbrevity of this description.

With reference to FIG. 2, an exemplary trajectory prediction system 170of FIG. 1 is illustrated. The trajectory prediction system 170 isimplemented to perform methods and other functions as disclosed hereinrelating to controlling the operation of vehicle 100 based, at least inpart, on past, current, observed, or predicted future trajectories ofthe vehicle 100 itself and/or based on past, current, observed, orpredicted trajectories of one or more road agents external to thevehicle 100. In some aspects, the trajectory of the vehicle 100 or aroad agent can be modeled in three-dimensional space.

The trajectory prediction system 170 is shown as including one or moreprocessors 110 from the vehicle 100 of FIG. 1. The one or moreprocessors 110 may be a part of the trajectory prediction system 170,the trajectory prediction system 170 may include one or more separateprocessors from the one or more processors 110 of the vehicle 100, orthe trajectory prediction system 170 may access the one or moreprocessors 110 through a data bus or another communication path,depending on the embodiment. In one aspect, the trajectory predictionsystem 170 includes a memory 172 that stores at least atrajectory-prediction module 174 and a control module 176. The memory172 may be a random-access memory (RAM), read-only memory (ROM), ahard-disk drive, a flash memory, or other suitable memory for storingthe modules 174, 176. The modules 174, 176 are, for example,computer-readable instructions that when executed by the one or moreprocessors 110, cause the one or more processors 110 to perform thevarious functions disclosed herein.

In connection with predicting the trajectory of the vehicle 100, thetrajectory prediction system 170 can store various kinds ofmodel-related data 178 in a database 180. As shown in FIG. 1, thetrajectory prediction system 170 may receive sensor data from a sensorsystem 120 in the vehicle 100 (the ego-vehicle). For example, in someaspects, the trajectory prediction system 170 receives image data fromone or more cameras 126. The trajectory prediction system 170 may alsoreceive LIDAR data from LIDAR sensors 124, radar data from radar sensors123, and/or sonar data from sonar sensors 125, depending on theparticular embodiment. In some aspects, the trajectory prediction system170 also receives inputs from vehicle systems 140. Examples include,without limitation, steering wheel angle, gas pedal (accelerator)position, linear velocity, and angular velocity. Steering-wheel-angleand gas-pedal-position data are examples of what may be termedcontroller-area-network (CAN bus) data, and linear velocity and angularvelocity are examples of what may be termed Inertial Measurement Unit(IMU) data. Certain of the above types of sensor data pertain topredicting the trajectory of vehicle 100 (the ego-vehicle) but not topredicting the trajectory of an external road agent, as explainedfurther below. As also indicated in FIG. 1, the trajectory predictionsystem 170, in particular the control module 176, can communicate withvehicle systems 140 and/or autonomous driving module(s) 160 to assistwith semi-autonomous or autonomous control over various functions of thevehicle 100. The control module 182 also includes instructions thatcause the one or more processors 110 to control the operation of theuser interface system 182 and coordinate the data, including predictedtrajectories, provided to various displays throughout the vehicle 100.

In some aspects, other or additional kinds of data from sensor system120 can be fed to the trajectory prediction system 170, such as radar,and/or sonar data. Additionally, more highly structured data such as arasterized map data (e.g., an occupancy grid for the environmentsurrounding vehicle 100) can be fed to a variational trajectorypredictor. The specific kinds of raw sensor data or structured data arefed to the trajectory prediction system 170 can vary, depending on theaspect.

In some aspects that include confidence scores, described below, theconfidence scores may be computed based, at least in part, on the numberof iterations that occur between the prediction of ego-vehicletrajectories and the prediction of road-agent trajectories while thepredicted ego-vehicle and road-agent trajectories are being iterativelyupdated. In general, a greater number of iterations corresponds to ahigher level of confidence in the resulting predicted trajectoriesbecause the predicted trajectories tend to converge to more stablepredictions after sufficient iterations.

As described in detail herein, the trajectory-prediction module 174generally includes instructions that cause the one or more processors110 to produce one or more predicted trajectories for the vehicle 100(the ego-vehicle) and one or more predicted trajectories for at leastone external road agent for display on a user interface. Various userinterface designs may be useful for displaying trajectory informationregarding the present technology, and the description provided herein isnot meant to limit the types of displays useful with the presenttechnology.

FIG. 3 provides a partial perspective view of an exemplary vehicleinterior compartment 50, illustrating two front seats for vehiclepassengers, as well as various vehicle controls. As can be seen, thevehicle includes a navigation display 52 and a head's up display (HUD)54 projected on a windshield 56 with multiple panels 58 that canaccommodate the display for a user interface. Multi-information displays(MIDs) 60, such as screens/displays that can toggle between differentinformational displays, can also be used, located in various areas ofthe vehicle interior compartment. In other aspects, personal electronicsdevices such as phones 62, tablets (not shown), and the like can also beused for display purposes. A number of variations in the architecturejust described are possible, depending on the particular aspect. Invarious aspects, the systems and methods provided herein may include theuse of a road agent provided as an automobile, a motorcycle, a bicycle,and/or a pedestrian; and the vehicle user interface is one of anavigation display, a multi-information display, a heads-up display(HUD), a head mounted display (HMD), a remote operator display, and awearable device. Multiple display may be used in combination with oneanother and may include different perspective points of view.

FIGS. 4A-4E are five example displays illustrating a user interfacehaving an image representing a top plan view 300 of an ego-vehicle 100and at least one road agent such as a vehicle 200 or pedestrian 204 withrespective trajectories 202, 206. FIGS. 5A-5E are five example displaysillustrating a user interface having an image representing a frontperspective view 310, or driver's view, of the same ego-vehicle 100 andthe road agent vehicle(s) 200 or pedestrians 202 and their respectivetrajectories 202, 204 in the same scenarios as provided in FIGS. 4A-4E,showing an overlap in trajectories and vehicles.

For example, FIG. 4A illustrates a top plan view 300 with two road agentvehicles 200. While the road agent predicted trajectories 202 do notoverlap in FIG. 4A, in FIG. 5A, with the driver's perspective point ofview 310, the road agent predicted trajectories 202 are close to oneanother, with portions overlapping that may cause confusion to a user.FIGS. 4B-4D illustrate top plans views 300 of an ego-vehicle 100 with anego-vehicle predicted trajectory 102 and a plurality of road agentvehicle 200 and their respective road agent predicted trajectories 202in different traffic patterns. As shown, the various predictedtrajectories 102, 202 not only overlap with one another, they alsooverlap certain of the road agent vehicles themselves, likely causingconfusion with a user. FIGS. 5B-5D illustrate driver's views 310 ofthose vehicles 100, 200 and the similar overlap of predictedtrajectories 102, 202, also providing a complex visualization likelyinterfering with visibility and/or causing confusion with a user. FIGS.4E and 5E illustrate the ego-vehicle 100 and predicted trajectory 102adjacent a cross-walk with a plurality of pedestrians 204 and theirrespective predicted trajectories 206, with various overlapping.

FIG. 6 is a flow chart diagram of a method 320 of modifying at least onepredicted trajectory based on an overlap of the predicted trajectorywith an object, in accordance with an illustrative aspect of the presenttechnology. The method 320 is for generating trajectories for a vehicleuser interface showing a driver's perspective view. The method firstincludes generating an ego-vehicle predicted trajectory 102 for anego-vehicle 100, and generating at least one road agent predictedtrajectory 202 for a road agent that is external to the ego-vehicle 100.The road agent(s) can be other vehicles 200, pedestrians 204, andcombinations thereof. As indicated by method step 322, after therespective predicted trajectories 102, 202 are generated, the method 320continues by determining that at least one road agent predictedtrajectory 202 exists behind an object when viewed in the driver'sperspective, indicating travel behind the object. As indicated by methodstep 324, the method continues to determine whether the predictedtrajectory 202 overlaps with the object, when displayed on the userinterface showing a driver's perspective view. Unless otherwiseindicated, the term “object”, as used with the methods described herein,can broadly include a static object, such as a parked vehicle, building,median divider, etc.; and/or can also include a moving object, which mayinclude a moving vehicle, a moving pedestrian, and the like. The method320 then includes modifying the at least one road agent predictedtrajectory 202 to remove the overlap, as indicated by method step 326.The method then proceeds with updating a display of the user interfaceto include any modified road agent predicted trajectory(s). With renewdreference to FIG. 2, in various aspects, a control module 176 can beused to provide instructions to one or more processors 110 and/or theuser interface system 182 to update a user interface to include adisplay of any modified road agent predicted trajectory.

FIG. 7 is a flow chart diagram of a method 328 of modifying at least onepredicted trajectory based on an overlap of the predicted trajectorywith an object, similar to the method 320 of FIG. 6, but with anadditional feature. As shown in FIG. 7, there is an additional methodstep 330 of determining whether the object is a road agent with its ownpredicted trajectory. If so, no modifications are made. If it is not asource, then the modifications are made.

FIGS. 8A-8C illustrate the modification of at least one predictedtrajectory based on an overlap with an object, in accordance with themethods of FIGS. 6-7. FIG. 8A is a top plan view 300 that provides tworoad agent vehicles 200A, 200B, each with a respective road agentpredicted trajectory 202A, 202B with an arrow to indicate a direction oftravel. The road agent vehicles 200A, 200B are travelling in oppositedirections. From the ego-vehicle driver's perspective point of view, onevehicle 200A will travel behind an object 208, and one vehicle 200B willtravel in front of the object 208. FIG. 8B provides the driver'sperspective point of view 310 of the situation as presented in FIG. 8A.As shown in FIG. 8B, the road agent predicted trajectory 202A may causeconfusion because it overlaps the object 208 and it appears that thevehicle 200A will travel in front of the object 208 while, in reality,the vehicle 200A will travel behind the object 208. FIG. 8C illustratesthe resulting modification of a length of the road agent predictedtrajectory 202A being shortened so as to no longer overlap with theobject 208.

FIGS. 9A-9C illustrate a modification of at least one road predictedtrajectory based on an overlap with two objects, in accordance with anillustrative aspect of the present technology. FIG. 9A is a top planview 300 that provides two road agent vehicles 200A, 200B, each with arespective road agent predicted trajectory 202A, 202B with an arrow toindicate a direction of travel. The road agent vehicles 200A, 200B aretravelling in opposite directions. From the ego-vehicle driver'sperspective point of view, one vehicle 200A will travel behind a firstobject 208, and one vehicle 200B will travel in front of the firstobject 208, but behind a second object 110. FIG. 9B provides thedriver's perspective point of view 310 of the situation as presented inFIG. 9A. As shown in FIG. 9B, the road agent predicted trajectories202A, 202B may cause confusion because they both overlap the objects208, 210 and it appears that both vehicles 200A, 200B will travel infront of the objects 208, 210 while, in reality, the vehicle 200A willtravel behind the first object 208, and the vehicle 200B will travel infront of the first object 208 and behind the second object 210. FIG. 9Cillustrates the resulting modification of a length of the both roadagent predicted trajectories 202A, 202B being shortened so as to nolonger overlap with the objects 208, 210.

FIG. 10 is flow chart diagram of a method 332 of modifying at least oneroad predicted trajectory based on an overlap with an object by hiding,diluting, and/or diluting the predicted trajectory, in accordance withvarious aspects of the present technology. The method step of 334determines whether a road agent predicted trajectory overlaps with anobject when displayed on a user interface with a driver's perspectivepoint of view. Similar to the method 320 of FIG. 8, this method 332 alsodetermines whether a road agent predicted trajectory exists partially orfully behind an object, indicating travel behind the object, asillustrated in method step 336. If yes, the method 332 proceeds bymodifying a display of the object and the predicted trajectories. Forexample, a portion or an entirety of the road agent predicted trajectorymay be modified so as to be hidden from, or behind, the object, asillustrated in method step 338. If not, the method 332 proceeds bymodifying a portion or entirety of the predicted trajectory to bediluted, blended, or the like, as illustrated in method step 340, inorder for the user to better understand the anteroposterior relationshipin the situation. In other aspects, the method includes instructions tomodify a display of the object such that a first part of the road agentpredicted trajectory appears hidden behind the object when displayed onthe user interface showing a driver's perspective view; and modify asecond part of the road agent predicted trajectory using at least onetechnique selected from the group consisting of hiding, diluting, andblending.

FIGS. 11A-11E illustrate various modifications of at least one roadagent predicted trajectory, in accordance with the methods 332 of FIG.10. FIG. 11A is a top plan view 300 that provides two road agentvehicles 200A, 200B, each with a respective road agent predictedtrajectory 202A, 202B with an arrow to indicate a direction of travel.The road agent vehicles 200A, 200B are travelling in oppositedirections. From the ego-vehicle driver's perspective point of view, onevehicle 200A will travel behind an object 208, and one vehicle 200B willtravel in front of the object 208. FIG. 8B provides the driver'sperspective point of view 310 of the situation as presented in FIG. 8A.As shown in FIG. 8B, the road agent predicted trajectory 202A may causeconfusion because it overlaps the object 208 and it appears that thevehicle 200A will travel in front of the object 208 while, in reality,the vehicle 200A will travel behind the object 208. FIG. 11C provides afirst resulting modification of having both road agent predictedtrajectories 202A, 202B being modified as to have a portion of each behidden behind the object 208. This provides a user with a more clearview of the object 208, however, there may be an ambiguity as to whetherthe second vehicle 200B is travelling in front of, or behind, the object208. FIGS. 11D and 11E may provide a user with a more clearunderstanding of the situation. In FIG. 11D, the road agent predictedtrajectory 202A for the road agent vehicle 200A travelling behind theobject is partially hidden behind the object 208, while the road agentpredicted trajectory 202B for the road agent vehicle 200B travelling infront of the object 208 is partially blended with the object 208. InFIG. 11E, the road agent predicted trajectory 202A for the road agentvehicle 200A travelling behind the object is partially diluted behindthe object 208, while the road agent predicted trajectory 202B for theroad agent vehicle 200B travelling in front of the object 208 ispartially blended with the object 208. Different combinations of hiding,blending, and diluting can be used.

FIG. 12 is a flow chart diagram of a method 342 of selecting differenttechniques of modifying at least one predicted trajectory based ondifferent overlaps, in accordance with various aspects of the presenttechnology. As illustrated in method step 344, the method determines theanteroposterior relationship between various road agents and objectsthat are displayed in a user interface display having a driver'sperspective point of view. The method then determines whether a targetobject overlaps with another object, as illustrated by method step 346.If there is an overlap the method determines whether another road agentreaches the overlapping point before the ego-vehicle will reach theoverlapping point. If yes, step 350 of the method directs the use of themethod as provided in FIG. 7 to modify the road agent predictedtrajectories. If not, step 352 of the method directs the use of themethod as provided in FIG. 10 to modify the road agent predictedtrajectories.

FIGS. 13A-13B illustrate modifications of at least one predictedtrajectory in accordance with the methods 342 of FIG. 12, while ignoringthe ego-vehicle trajectory for simplicity. FIG. 13A is a top plan view300 including three road agent vehicles 200A, 200B, 200C in series, eachwith a respective predicted trajectory 202A, 202B, 202C. Two of thepredicted trajectories 202A, 202B overlap with an adjacent road agentvehicle 200B, 200C. FIG. 13B provides a driver's perspective point ofview 310 after implementing the method 342 of FIG. 12. For example,since there is no road agent or object in advance of vehicle 200C, themethod of FIG. 7 is applied to the road agent predicted trajectory 202C,which is provided in the display. Since there are other vehicles infront of both vehicles 200A and 200B, the method of FIG. 10 is appliedto their respective predicted trajectories 202A, 202B. As a result,predicted trajectory 202A overlaps with the vehicle 200B, but thepredicted trajectory 202B is hidden due to the overlap with vehicle200C.

FIG. 14 is a flow chart diagram of a method 354 of modifying at leastone predicted trajectory based on an overlap of at least two projectedtrajectories in a driver's perspective view, with an optional use ofconfidence scores, in accordance with an illustrative aspect of thepresent technology. The method 354 includes generating an ego-vehiclepredicted trajectory for an ego-vehicle; and generating at least oneroad agent predicted trajectory for a road agent that is external to theego-vehicle. After the predicted trajectories are generated, the methodcontinues by determining that at least two predicted trajectoriesoverlap when displayed on the user interface showing a driver'sperspective view, when displayed on the user interface showing adriver's perspective view, as shown by method step 356. In variousaspects, the method 356 may proceed directly to step 364, which includesmodifying the at least one road agent predicted trajectory to remove theoverlap. The method then proceeds with updating a display of the userinterface to include any modified road agent predicted trajectory. Inoptional methods as shown in method step 358, the trajectory-predictionmodule 170 may include instructions to calculate or otherwise obtain aconfidence score that is representative of a likelihood of a collisionbetween road agents due to the presentation of an overlap of road agentpredicted trajectories. The methods may also include performing acomparison of the confidence score in order to determine that theconfidence score is less than a predetermined threshold, as indicated bymethod step 360. Once it is determined that the risk of a collision isless than a predetermined threshold, the method optionally continues instep 362 by determining whether another road agent will reach theoverlapping point of the predicted trajectories before the ego-vehiclewill reach that intersection. If yes, the method includes modifying atleast one predicted trajectory in order to remove the overlap, as shownby method step 364. In various aspects, the modification may include ashortening of a length of the predicted trajectory, providing aseparation distance between at least two of the predicted trajectories,and modifying a predicted trajectory using a technique such as hiding,diluting, blending, or similarly modifying at least a portion of thepredicted trajectory (and/or adjacent road agent or object).

FIG. 15 is a flow chart diagram of a method 366 of modifying at leastone predicted trajectory based on the calculation of a shorted distancebetween two adjacent predicted trajectories being below a thresholdvalue, in accordance with an illustrative aspect of the presenttechnology. For example, the trajectory-prediction module 170 mayinclude an instruction to generate an ego-vehicle predicted trajectoryfor an ego-vehicle, and to generate at least one road agent predictedtrajectory for a road agent that is external to the ego-vehicle. Asshown in method steps 368 and 370, the instructions may include a stepto determine that a distance between two adjacent predicted trajectoriesis below a predetermined threshold value when displayed on the userinterface showing a driver's perspective view. Thereafter, theinstructions may include a step to perform at least one modificationselected from the group consisting of: (1) change a color of at leastone of the road agent predicted trajectories; (2) change a spacinglocation of at least one of the road agent predicted trajectories; (3)change a thickness of at least one of the road agent predictedtrajectories; and (4) determine a priority road agent predictedtrajectory based on a closest proximity to the ego-vehicle, and onlydisplay the priority road agent predicted trajectory. A control module176 may also provide instructions that, when executed by the one or moreprocessors 110, cause the one or more processors 110, or a userinterface system 182, to update the user interface to include anymodified road agent predicted trajectory.

FIGS. 16A-16C illustrate modifications of at least one road agent andego-vehicle predicted trajectory, in accordance with the methods 366 ofFIG. 14. FIG. 16A illustrates a top plan view 300 of an ego-vehicle 100with its predicted trajectory 102, as well as two road agent vehicles200A, 200B and their respective predicted trajectories 202A, 202B. Asshown, the ego-vehicle predicted trajectory 102 overlaps the road agentpredicted trajectory 202B, and the other road agent predicted trajectory202A overlaps with the ego-vehicle 100. FIG. 16B provides a driver'sperspective point of view 310 that completely hides (removes) the roadagent vehicle 200A and its predicted trajectory 202A, and shortens alength of the ego-vehicle predicted trajectory 102 to remove the overlapof predicted trajectories 102, 202B. FIG. 16C provides a driver'sperspective point of view 310 that completely hides (removes) the roadagent vehicle 200A and its predicted trajectory 202A, and blends a colorof the ego-vehicle predicted trajectory 102 to minimize the presence ofthe overlap of predicted trajectories 102, 202B.

FIG. 17 illustrates a user interface display from a driver's perspectiveview 310 with an overlap in road agents 200A, 200B with predictedtrajectories 202A, 202B. FIGS. 18A-18E illustrate modifications of atleast one predicted trajectory of FIG. 17, in accordance with themethods of FIGS. 14-15. For example, in FIGS. 18A-18B, a color,gradient, or pattern of one road agent predicted trajectory 202A can bechanged to make it appear distinct from another road agent predictedtrajectory 202B. In FIG. 18C, the road agent predicted trajectories202A, 202B can be provided with different thicknesses. In FIG. 18D, oneof the predicted trajectories 202A can be shifted or separated by adistance “a” in order to provide more spacing between the adjacentpredicted trajectories 202A, 202B. In FIG. 18E, one of the predictedtrajectories 202A can be removed (hidden) from view altogether.

FIG. 19 is a flow chart diagram of a method 374 of modifying at leastone predicted trajectory based on an overlap of either at least twoprojected trajectories, or one or more projected trajectory and one ormore object, with a determination of a priority intersection, inaccordance with an illustrative aspect of the present technology. Aftergenerating the required predicted trajectories, the method includesdetermining whether at least two of the predicted trajectories (ortrajectory and object) overlap at a first intersection, as shown bymethod step 376, and determining whether at least two of the predictedtrajectories (or trajectory and object) overlap at a secondintersection, as shown by method step 378. When at least two overlapsare located, method step 380 provides for the determination of apriority intersection. This determination is based on a calculation ofwhich road agents will first arrive at one of the first and secondintersections. Once the priority intersection is located, the methodincludes modifying at least one predicted trajectory that overlaps withthe priority intersection, as shown by method step 382.

To further explain the method 374 of FIG. 19, FIG. 20 is provided asillustrating a set of four vehicles having predicted trajectoriesoverlapping at four intersections having various intersection times.FIG. 20 includes four intersections labeled as A, B, C, and D. Thepriority intersection here is at intersection B, which will occur firstin time, assuming the vehicles are travelling at the same velocity.Intersection C should occur last in time, with intersections A and Doccurring at some time between between B and C.

FIGS. 21A-21B further illustrate the modifications of at least onepredicted trajectory, in accordance with the methods of FIG. 19. FIG.21B is a top plan view 300 of an ego vehicle 100, a road agent vehicle200, and two pedestrians 204, each with a respective predictedtrajectory 102, 202, 206A, 206B. There are two stop signs 214 providedat the roadway intersection. Since the road agent vehicle 200 is at astop sign, the priority intersection is between the ego-vehicle 100 andone of the pedestrians 204A, 204B, depending on speed of thepedestrians. The last intersection in time will be between theego-vehicle 100 and the road agent vehicle 200. FIG. 21A provides adriver's perspective view 310 of the situation presented in FIG. 21A,and has modified the ego-vehicle predicted trajectory 102 by shorteninga length thereof because it is involved with the priority intersection.

FIG. 22 is a flow chart diagram of a method 384 of modifying at leastone predicted trajectory based on an overlap of a projected trajectoryand an object, with a determination that the object is a partially orfully hidden road agent, in accordance with an illustrative aspect ofthe present technology. After generating the required predictedtrajectories, the method includes a step 386 of determining that a roadagent predicted trajectory exists behind an object, indicating travelbehind the object when considering a driver's perspective point of view.As shown by method step 388, the method determines that the road agentpredicted trajectory overlaps with the objected, when displayed on auser interface. If it is determined in method step 390 that at least onehidden road agent is behind the object when displayed on the userinterface showing a driver's perspective view, step 392 providesperforming at least one modification selected from the group consistingof: removing both the hidden road agent and the respective road agentpredicted trajectory; hiding, diluting, or blending a portion of theroad agent predicted trajectory with the object; and superimposing thehidden road agent over the object.

FIGS. 23A-23D illustrate example situations with a requirement for amodification of at least one predicted trajectory, in accordance withthe methods of FIG. 22. For example, FIG. 23A provides a top plan view300 including a smaller vehicle 200A located adjacent a larger vehicle200B, such as a truck. FIG. 23B illustrates a top plan view 300including a vehicle 200 adjacent an object, such as a building 212.FIGS. 23C and 23D provide driver's perspective views 310 of thesituations presented in FIGS. 23A and 23B, respectively, in which theroad agent vehicle is not in view. FIGS. 24A-24D illustrate additionalmodifications of at least one predicted trajectory, in accordance withthe methods of FIG. 22 when presented with the situations of FIGS. 23Cand 23D. FIG. 24A completely removes both the hidden road agent vehicleand the respective road agent predicted trajectory for both situations.FIG. 24B removes the hidden road agent but still presents the predictedtrajectories. FIG. 24C blends the predicted trajectories with the roadagent vehicle 200B and the building 212. FIG. 24D superimposes thehidden road agents 200, 200A with the building 212 and road agentvehicle 200B, respectively.

FIG. 25 is a flow chart diagram of a method 394 of modifying at leastone predicted trajectory based on the presence of a sloped terrain in aroadway, in accordance with an illustrative aspect of the presenttechnology. The method 394 is for generating trajectories for a vehicleuser interface showing a driver's perspective view, and includesgenerating an ego-vehicle predicted trajectory for an ego-vehicle, andgenerating at least one road agent predicted trajectory for a road agentthat is external to the ego-vehicle. As indicated by method step 396,after the respective predicted trajectories are generated, the method394 continues by determining that at least one road agent predictedtrajectory exists behind an object when viewed in the driver'sperspective, indicating travel behind the object. The method furtherdetermines that the road agent predicted trajectory overlaps with theobject when displayed in the driver's perspective point of view.Specifically in this method aspect, there is a determination that theobject is a sloped terrain, and the predicted trajectory is at leastpartially hidden, as indicated by method step 400. As shown in FIG. 26A,the sloped terrain 214 can be uphill or downhill. As indicated by methodstep 402, the method continues to modifying at least one road agentpredicted trajectory for display as a curved trajectory, extending overthe sloped terrain such that it is visible when displayed on the userinterface showing the driver's perspective point of. The method thenproceeds with updating a display of the user interface to include anymodified road agent predicted trajectory(s).

FIGS. 26A-26D and 27A-27D illustrate modifications of at least onepredicted trajectory where a sloped terrain interferes with the view, inaccordance with the methods of FIG. 25. FIG. 26A is a partialperspective view of an example scenario with an ego-vehicle 100 and aroad agent vehicle 200 with its predicted trajectory 202. FIG. 26Bprovides a side plan view of the differences in elevation, specificallyshowing the sloped terrain 214 and with the road agent predictedtrajectory 202 pointing in an upward direction due to the slope, whichis not desirable. FIG. 26C provides a top plan view 300, while FIG. 26Dprovides the driver's perspective point of view 310 that has a hiddenroad agent vehicle 200 and only a portion of the predicted trajectory202 showing, likely causing confusion for the user as to both thelocation of the road agent as well as its direction of travel. FIGS. 27Aand 27B illustrate how the predicted trajectory of the road agentvehicle 200 is transformed from a straight line 202 to a curved, or atleast partially curved, trajectory 203 that can extend a substantiallyfixed distance away from (over) the sloped terrain such that it does notindicate a direction that appears to be leading up into the sky or downinto the roadway. For example, if the distance between a trajectory 202and the road/terrain is increasingly larger than a predeterminedthreshold, indicating upward travel, at least a portion of the predictedtrajectory can be curved 203 in a downward direction, as shown in FIG.27A. In another example, if the distance between a trajectory 202 andthe road/terrain is decreasingly smaller than another predeterminedthreshold, at least a portion of the predicted trajectory can be curved203 in an upward direction, as shown in FIG. 27B. FIG. 27C illustrates adriver's perspective point of view 310 of the scenario of FIG. 27A, withthe curved predicted trajectory 203. FIG. 27D further provides asuperimposed representation of the road agent vehicle 200, which may beprovided with a color and/or shape to indicate that is hidden, locatedat the other side of a hill, or the like.

FIG. 28 is a flow chart diagram of a method 404 of modifying at leastone predicted trajectory based on at least one road agent predictedtrajectory protruding from a display area, which would otherwise providea user with an unknown direction of travel of the road agent, inaccordance with an illustrative aspect of the present technology. Themethod 404 is for generating trajectories for a vehicle user interfaceshowing a driver's perspective view, and includes generating anego-vehicle predicted trajectory for an ego-vehicle, and generating atleast one road agent predicted trajectory for a road agent that isexternal to the ego-vehicle. As indicated by method step 406, after therespective predicted trajectories are generated, the method 404 may usethe trajectory-prediction module having an instruction to determine thatat least one road agent predicted trajectory protrudes from a displayarea, and further has an unknown direction of travel when displayed onthe user interface showing a driver's perspective view, as shown bymethod step 408. The methods may include a step 410 to modify the atleast one road agent predicted trajectory to provide an indication ofdirection in the display area.

FIGS. 29A-29E illustrate a modification of at least one predictedtrajectory, in accordance with the methods of FIG. 28. FIG. 29Aillustrates a top plan view 300 with an ego-vehicle 100 and a road agentvehicle 200. Both the road agent vehicle 200 and the arrow of directionextend a distance off from the display. FIG. 29B illustrates thisscenario from a driver's perspective point of view 310. As shown, only aportion of the road agent predicted trajectory 202 is present, whichleaves confusion for the user with respect to the direction of travel.FIG. 29C adds an icon representative of at least a partial view of theroad agent vehicle 200, however this may still be confusing to a userdue to the lack of a directional arrow. FIGS. 29D and 29E modify alength of the road agent predicted trajectory 202, and add a directionalarrow to the appropriate end of the predicted trajectory in order toprovide the additional information to a user when viewed from a driver'sperspective point of view.

FIG. 30 is a flow chart diagram of a method 412 of modifying at leastone predicted trajectory based on an overlap of a projected trajectoryand either an object or another predicted trajectory, with adetermination that the predicted trajectory overlaps one or more staticobjects, in accordance with an illustrative aspect of the presenttechnology. The method 412 is for generating trajectories for a vehicleuser interface showing a driver's perspective view, and includesgenerating an ego-vehicle predicted trajectory for an ego-vehicle, andgenerating at least one road agent predicted trajectory for a road agentthat is external to the ego-vehicle. As indicated by method step 414,after the respective predicted trajectories are generated, the method412 continues by determining that at least one road agent predictedtrajectory overlaps either an object or another road agent predictedtrajectory. The method continues with step 416 to determine whether theroad agent predicted trajectory overlaps one or more static objects, inparticular, static objects that do not reside on a roadway or on asidewalk. Non-limiting examples of such static objects may include a rowof trees, a series of buildings or structures, etc. A plurality ofstatic objects may be considered as a group or individually, dependingon their size and location. As shown in method step 418, the methodincludes modifying the road agent predicted trajectory by hiding,diluting, or blending the road agent predicted trajectory at one or moreof the locations where the predicted trajectory overlaps one or more ofthe static objects.

FIGS. 31A-31C illustrate a modification of at least one predictedtrajectory that is adjacent a plurality of static objects, in accordancewith the methods of FIG. 30. FIG. 31A provides a top plan view 300 of anego-vehicle 100 and a road agent vehicle 200. The predicted trajectory202 is in front of a plurality of static objects 216. When presented inthe driver's perspective point of view 310 as shown in FIG. 31B, therelationship between the road agent predicted trajectory 202 and thestatic objects 216 may appear confusing. Accordingly, as shown in FIG.31C, the road agent predicted trajectory 202 is modified such atareas/locations of the predicted trajectory 202 that overlap with theobjects 216 are provided as blended with the objects 216. In otheraspects, those portions may be diluted and/or hidden, depending on thetype of static object and optionally other factors.

FIG. 32 is a flow chart diagram of a method 420 of modifying at leastone predicted trajectory based on an overlap of a projected trajectoryand either an object or another predicted trajectory, with adetermination that the predicted trajectory is a past tense road agenttrajectory, in accordance with an illustrative aspect of the presenttechnology. Typically the road agent predicted trajectories are currentand/or future trajectories. However, in various aspects it may bebeneficial or desirable to provide at least one past tense trajectory,or an indication of a road agent trajectory that has already traversed aportion of the roadway and no longer provides a threat of collision withthe ego-vehicle. The method 420 is for generating trajectories for avehicle user interface showing a driver's perspective view, and includesgenerating an ego-vehicle predicted trajectory for an ego-vehicle, andgenerating at least one road agent predicted trajectory for a road agentthat is external to the ego-vehicle. As indicated by method step 422,after the respective predicted trajectories are generated, the method420 continues by determining that at least one road agent predictedtrajectory overlaps either an object or another road agent predictedtrajectory. The method continues with step 424 to determine that theroad agent predicted trajectory is actually a past tense road agentpredicted trajectory. In other words, the travel across the roadway hasalready occurred. As shown in method step 426, the method includesmodifying the past tense road agent predicted trajectory, for example,by modifying or changing a shape or thickness thereof.

FIGS. 33A-33C illustrate a modification of at least one predictedtrajectory, in accordance with the methods of FIG. 32. For example, FIG.33A provides a top plan view 300 of an ego-vehicle 100 and a first roadagent vehicle 200A having a current or future trajectory 202A and asecond road agent vehicle 200B with a past tense road agent predictedtrajectory 202B. FIG. 33B provides a driver's perspective point of view310 of the scenario as provided in FIG. 33A. The ego-vehicle predictedtrajectory is omitted for clarity. However, the presence of the pasttense road agent predicted trajectory 202B may be confusing for user.Thus, FIG. 33C provides a modification of the shape and/or thickness,etc., of the past tense road agent predicted trajectory.

FIG. 34 is a flow chart diagram of a method 428 for selecting a type ofdisplay to be provided in a user interface based on complexity, inaccordance with an illustrative aspect of the present technology. Themethod 428 is for generating trajectories for a vehicle user interfaceshowing a driver's perspective view, and includes generating anego-vehicle predicted trajectory for an ego-vehicle, and generating atleast one road agent predicted trajectory for a road agent that isexternal to the ego-vehicle. As indicated by method step 430, after therespective predicted trajectories are generated, the method 428continues by determining that at least one road agent predictedtrajectory overlaps either an object or another road agent predictedtrajectory. The method continues with step 432 where the trajectoryprediction module determines that a display in the user interfaceshowing the perspective view may be complex or confusing whenpresented/displayed with a driver's perspective field of view. Invarious aspects, a determination of complexity can be based on one ormore of many factors, including: a threshold number and/or type of roadagents and objects present in the display; locations of the road agents,objects, and predicted trajectories; time of day; traffic congestion;weather; experience of the user; duration of travel; and the like. Invarious aspects, a determination of complexity can be based on one ormore calculations or confidence scores that may be based on, forexample, density of the predicted trajectories, a number of road agentsand/or objects, and a number of overlapping points. As provided inmethod step 434, the method includes generating a display in the userinterface showing a top plan view, as opposed to a driver's perspectivepoint of view. The use of the top plan view is intended to simplify thedisplay and provide a user with a more complete display of surroundingsand scenarios. In various aspects, the methods may include requestinginstructions to obtain a selection request from a user, and allow a userto preview different display options and to freely switch between thedifferent views. In various other aspects, the methods may includeproviding both types of displays to the user, for example, providing atop plan view in a first display, and a driver's perspective point ofview display in a second display. In still further aspects, a singledisplay can be generated providing both a top plan view and a driver'sperspective point of view in a side-by-side arrangement, or the like.

With respect to the display of icons as well as the depiction ofpredicted trajectories, the roadways, objects, etc., it should beunderstood that the present technology should not be limited to thespecific types and styles that are specifically described herein, andthey may be customized as desired. In this regard, 35A-35C illustratedifferent non-limiting variations of predicted trajectories havingdifferent lines, 2-D patterns, and 3-D shapes. For example, FIG. 35provides different types of line and arrow combinations and designs,which may include indications (shown as dots in FIG. 35A) of separatesegments that may indicate a change in direction of travel. FIG. 35Bprovides different types of two dimensional shapes that may be used inthe place of the lines of FIG. 35. The two-dimensional shapes may bedesigned with different gradients or colors to better indicate adirection of travel, speed, confidence levels, and the like. FIG. 35Cprovides different types of three-dimensional shapes that may be used inthe place of the lines of FIG. 35. The three-dimensional shapes may besimilarly be designed with different gradients or colors to betterindicate a direction of travel, speed, confidence levels, and the like.The three-dimensional shapes may be provided with various levels ofdetail, ranging from the use of lines and simple shapes, to the displayof detailed three-dimensional objects.

FIGS. 36A-36C illustrate 2-D and 3-D predicted trajectories with anoverlap of two predicted trajectories. FIG. 36A illustrates the scenariopreviously discussed with respect to FIGS. 16A-16C, where theego-vehicle predicted trajectory 102 intersects and overlaps with theroad agent predicted trajectory 202. FIG. 36B provides two sets oftwo-dimensional shapes with varying colors and gradients to representthe predicted trajectories 102, 202. FIG. 36C provides two sets ofthree-dimensional shapes with varying colors and gradients to representthe predicted trajectories 102, 202.

FIGS. 37A-37C illustrate 2-D and 3-D predicted trajectories with anoverlap of a predicted trajectory and at least one static object. FIG.37A illustrates the scenario previously discussed with respect to FIGS.31A-31C, where the road agent vehicle 200 is travelling adjacent aplurality of static objects 216. FIG. 37B provides two-dimensionalshapes with varying colors and gradients to represent the predictedtrajectory 202. FIG. 37C provides three-dimensional shapes with varyingblending, diluting, and hiding to represent the predicted trajectory202.

FIGS. 38A-38B illustrate the use of a combination of 2-D and 3Dpredicted trajectories on a single display. FIG. 38A provides a top planview 300, and FIG. 38B provides a driver's perspective point of view 310of the scenario presented in FIG. 38A. Specifically, FIG. 38B providesthe ego-vehicle predicted trajectory 102 with a simple line pattern,provides a first road agent predicted trajectory 202A with athree-dimensional shape, and provides a second road agent predictedtrajectory 202B with a two-dimensional shape. It should be understoodthat various modifications and combinations can be used with the presenttechnology. In various aspects, the user may customize and change thedisplay types, and the systems and methods may include changing a shapeand/or dimension or predicted trajectories based on predeterminedthresholds and requirements.

Each of the various methods described herein can be provided as part ofsystems that can include one or more processors and memory that mayinclude a trajectory-prediction module including instructions that, whenexecuted by the one or more processors, cause the processors to executeactions to perform the steps described in various parts of the methods.Similarly, each of the methods described herein can be stored asinstructions on a non-transitory computer-readable medium.

FIG. 1 will now be discussed in full detail as an example vehicleenvironment within which the system and methods disclosed herein mayoperate. In some instances, the vehicle 100 is configured to switchselectively between an autonomous mode, one or more semi-autonomousoperational modes, and/or a manual mode. Such switching also referred toas handover when transitioning to a manual mode can be implemented in asuitable manner, now known or later developed. “Manual mode” means thatall of or a majority of the navigation and/or maneuvering of the vehicleis performed according to inputs received from a user (e.g., humandriver/operator).

In one or more aspects, the vehicle 100 is an autonomous vehicle. Asused herein, “autonomous vehicle” refers to a vehicle that operates inan autonomous mode. “Autonomous mode” refers to navigating and/ormaneuvering the vehicle 100 along a travel route using one or morecomputing systems to control the vehicle 100 with minimal or no inputfrom a human driver/operator. In one or more aspects, the vehicle 100 ishighly automated or completely automated. In one aspect, the vehicle 100is configured with one or more semi-autonomous operational modes inwhich one or more computing systems perform a portion of the navigationand/or maneuvering of the vehicle along a travel route, and a vehicleoperator (i.e., driver) provides inputs to the vehicle to perform aportion of the navigation and/or maneuvering of the vehicle 100 along atravel route. Thus, in one or more aspects, the vehicle 100 operatesautonomously according to a particular defined level of autonomy. Forexample, the vehicle 100 can operate according to the Society ofAutomotive Engineers (SAE) automated vehicle classifications 0-5. In oneaspect, the vehicle 100 operates according to SAE level 2, whichprovides for the autonomous driving module 160 controlling the vehicle100 by braking, accelerating, and steering without operator input butthe driver/operator is to monitor the driving and be vigilant and readyto intervene with controlling the vehicle 100 if the autonomous module160 fails to properly respond or is otherwise unable to adequatelycontrol the vehicle 100.

The vehicle 100 can include one or more processors 110. In one or morearrangements, the processor(s) 110 can be a main processor of thevehicle 100. For instance, the processor(s) 110 can be an electroniccontrol unit (ECU). The vehicle 100 can include one or more data stores115 for storing one or more types of data. The data store 115 caninclude volatile and/or non-volatile memory. Examples of suitable datastores 115 include RAM (Random Access Memory), flash memory, ROM (ReadOnly Memory), PROM (Programmable Read-Only Memory), EPROM (ErasableProgrammable Read-Only Memory), EEPROM (Electrically ErasableProgrammable Read-Only Memory), registers, magnetic disks, opticaldisks, hard drives, or any other suitable storage medium, or anycombination thereof. The data store 115 can be a component of theprocessor(s) 110, or the data store 115 can be operably connected to theprocessor(s) 110 for use thereby. The term “operably connected,” as usedthroughout this description, can include direct or indirect connections,including connections without direct physical contact.

In one or more arrangements, the one or more data stores 115 can includemap data 116. The map data 116 can include maps of one or moregeographic areas. In some instances, the map data 116 can includeinformation or data on roads, traffic control devices, road markings,structures, features, and/or landmarks in the one or more geographicareas. The map data 116 can be in any suitable form. In some instances,the map data 116 can include aerial views of an area. In some instances,the map data 116 can include ground views of an area, including360-degree ground views. The map data 116 can include measurements,dimensions, distances, and/or information for one or more items includedin the map data 116 and/or relative to other items included in the mapdata 116. The map data 116 can include a digital map with informationabout road geometry. The map data 116 can be high quality and/or highlydetailed.

In one or more arrangement, the map data 116 can include one or moreterrain maps 117. The terrain map(s) 117 can include information aboutthe ground, terrain, roads, surfaces, and/or other features of one ormore geographic areas. The terrain map(s) 117 can include elevation datain the one or more geographic areas. The map data 116 can be highquality and/or highly detailed. The terrain map(s) 117 can define one ormore ground surfaces, which can include paved roads, unpaved roads,land, and other things that define a ground surface.

In one or more arrangement, the map data 116 can include one or morestatic obstacle maps 118. The static obstacle map(s) 118 can includeinformation about one or more static obstacles located within one ormore geographic areas. A “static obstacle” is a physical object whoseposition does not change or substantially change over a period of timeand/or whose size does not change or substantially change over a periodof time. Examples of static obstacles include trees, buildings, curbs,fences, railings, medians, utility poles, statues, monuments, signs,benches, furniture, mailboxes, large rocks, hills. The static obstaclescan be objects that extend above ground level. The one or more staticobstacles included in the static obstacle map(s) 118 can have locationdata, size data, dimension data, material data, and/or other dataassociated with it. The static obstacle map(s) 118 can includemeasurements, dimensions, distances, and/or information for one or morestatic obstacles. The static obstacle map(s) 118 can be high qualityand/or highly detailed. The static obstacle map(s) 118 can be updated toreflect changes within a mapped area.

The one or more data stores 115 can include sensor data 119. In thiscontext, “sensor data” means any information about the sensors that thevehicle 100 is equipped with, including the capabilities and otherinformation about such sensors. As will be explained below, the vehicle100 can include the sensor system 120. The sensor data 119 can relate toone or more sensors of the sensor system 120. As an example, in one ormore arrangements, the sensor data 119 can include information on one ormore LIDAR sensors 124 of the sensor system 120.

In some instances, at least a portion of the map data 116 and/or thesensor data 119 can be located in one or more data stores 115 locatedonboard the vehicle 100. Alternatively, or in addition, at least aportion of the map data 116 and/or the sensor data 119 can be located inone or more data stores 115 that are located remotely from the vehicle100.

As noted above, the vehicle 100 can include the sensor system 120. Thesensor system 120 can include one or more sensors. “Sensor” means anydevice, component and/or system that can detect, and/or sense something.The one or more sensors can be configured to detect, and/or sense inreal-time. As used herein, the term “real-time” means a level ofprocessing responsiveness that a user or system senses as sufficientlyimmediate for a particular process or determination to be made, or thatenables the processor to keep up with some external process.

In arrangements in which the sensor system 120 includes a plurality ofsensors, the sensors can function independently from each other.Alternatively, two or more of the sensors can work in combination witheach other. In such a case, the two or more sensors can form a sensornetwork. The sensor system 120 and/or the one or more sensors can beoperably connected to the processor(s) 110, the data store(s) 115,and/or another element of the vehicle 100 (including any of the elementsshown in FIG. 1). The sensor system 120 can acquire data of at least aportion of the external environment of the vehicle 100 (e.g., nearbyvehicles).

The sensor system 120 can include any suitable type of sensor. Variousexamples of different types of sensors will be described herein.However, it will be understood that the aspects are not limited to theparticular sensors described. The sensor system 120 can include one ormore vehicle sensors 121. The vehicle sensor(s) 121 can detect,determine, and/or sense information about the vehicle 100 itself. In oneor more arrangements, the vehicle sensor(s) 121 can be configured todetect, and/or sense position and orientation changes of the vehicle100, such as, for example, based on inertial acceleration. In one ormore arrangements, the vehicle sensor(s) 121 can include one or moreaccelerometers, one or more gyroscopes, an inertial measurement unit(IMU), a dead-reckoning system, a global navigation satellite system(GNSS), a global positioning system (GPS), a navigation system 147,and/or other suitable sensors. The vehicle sensor(s) 121 can beconfigured to detect, and/or sense one or more characteristics of thevehicle 100. In one or more arrangements, the vehicle sensor(s) 121 caninclude a speedometer to determine a current speed of the vehicle 100.

Alternatively, or in addition, the sensor system 120 can include one ormore environment sensors 122 configured to acquire, and/or sense drivingenvironment data. “Driving environment data” includes and data orinformation about the external environment in which an autonomousvehicle is located or one or more portions thereof. For example, the oneor more environment sensors 122 can be configured to detect, quantifyand/or sense obstacles in at least a portion of the external environmentof the vehicle 100 and/or information/data about such obstacles. Suchobstacles may be stationary objects and/or dynamic objects. The one ormore environment sensors 122 can be configured to detect, measure,quantify and/or sense other things in the external environment of thevehicle 100, such as, for example, lane markers, signs, traffic lights,traffic signs, lane lines, crosswalks, curbs proximate the vehicle 100,off-road objects, etc.

Various examples of sensors of the sensor system 120 will be describedherein. The example sensors may be part of the one or more environmentsensors 122 and/or the one or more vehicle sensors 121. Moreover, thesensor system 120 can include operator sensors that function to track orotherwise monitor aspects related to the driver/operator of the vehicle100. However, it will be understood that the aspects are not limited tothe particular sensors described.

As an example, in one or more arrangements, the sensor system 120 caninclude one or more radar sensors 123, one or more LIDAR sensors 124,one or more sonar sensors 125, and/or one or more cameras 126. In one ormore arrangements, the one or more cameras 126 can be high dynamic range(HDR) cameras, infrared (IR) cameras and so on. In one aspect, thecameras 126 include one or more cameras disposed within a passengercompartment of the vehicle for performing eye-tracking on theoperator/driver in order to determine a gaze of the operator/driver, aneye track of the operator/driver, and so on.

The vehicle 100 can include an input system 130. An “input system”includes any device, component, system, element or arrangement or groupsthereof that enable information/data to be entered into a machine. Theinput system 130 can receive an input from a vehicle passenger (e.g. adriver or a passenger). The vehicle 100 can include an output system135. An “output system” includes any device, component, or arrangementor groups thereof that enable information/data to be presented to avehicle passenger (e.g. a person, a vehicle passenger, etc.).

The vehicle 100 can include one or more vehicle systems 140. Variousexamples of the one or more vehicle systems 140 are shown in FIG. 1.However, the vehicle 100 can include more, fewer, or different vehiclesystems. It should be appreciated that although particular vehiclesystems are separately defined, each or any of the systems or portionsthereof may be otherwise combined or segregated via hardware and/orsoftware within the vehicle 100. The vehicle 100 can include apropulsion system 141, a braking system 142, a steering system 143,throttle system 144, a transmission system 145, a signaling system 146,and/or a navigation system 147. Each of these systems can include one ormore devices, components, and/or combination thereof, now known or laterdeveloped.

The navigation system 147 can include one or more devices, sensors,applications, and/or combinations thereof, now known or later developed,configured to determine the geographic location of the vehicle 100and/or to determine a travel route for the vehicle 100. The navigationsystem 147 can include one or more mapping applications to determine atravel route for the vehicle 100. The navigation system 147 can includea global positioning system, a local positioning system or a geolocationsystem.

The processor(s) 110, the trajectory prediction system 170, and/or theautonomous driving module(s) 160 can be operably connected tocommunicate with the various vehicle systems 140 and/or individualcomponents thereof. For example, returning to FIG. 1, the processor(s)110 and/or the autonomous driving module(s) 160 can be in communicationto send and/or receive information from the various vehicle systems 140to control the movement, speed, maneuvering, heading, direction, etc. ofthe vehicle 100. The processor(s) 110, the trajectory prediction system170, and/or the autonomous driving module(s) 160 may control some or allof these vehicle systems 140 and, thus, may be partially or fullyautonomous.

The processor(s) 110, the trajectory prediction system 170, and/or theautonomous driving module(s) 160 can be operably connected tocommunicate with the various vehicle systems 140 and/or individualcomponents thereof. For example, returning to FIG. 1, the processor(s)110, the trajectory prediction system 170, and/or the autonomous drivingmodule(s) 160 can be in communication to send and/or receive informationfrom the various vehicle systems 140 to control the movement, speed,maneuvering, heading, direction, etc. of the vehicle 100. Theprocessor(s) 110, the trajectory prediction system 170, and/or theautonomous driving module(s) 160 may control some or all of thesevehicle systems 140.

The processor(s) 110, the trajectory prediction system 170, and/or theautonomous driving module(s) 160 may be operable to control thenavigation and/or maneuvering of the vehicle 100 by controlling one ormore of the vehicle systems 140 and/or components thereof. For instance,when operating in an autonomous mode, the processor(s) 110, thetrajectory prediction system 170, and/or the autonomous drivingmodule(s) 160 can control the direction and/or speed of the vehicle 100.The processor(s) 110, the trajectory prediction system 170, and/or theautonomous driving module(s) 160 can cause the vehicle 100 to accelerate(e.g., by increasing the supply of fuel provided to the engine),decelerate (e.g., by decreasing the supply of fuel to the engine and/orby applying brakes) and/or change direction (e.g., by turning the fronttwo wheels). As used herein, “cause” or “causing” means to make, force,compel, direct, command, instruct, and/or enable an event or action tooccur or at least be in a state where such event or action may occur,either in a direct or indirect manner.

The vehicle 100 can include one or more actuators 150. The actuators 150can be any element or combination of elements operable to modify, adjustand/or alter one or more of the vehicle systems 140 or componentsthereof responsive to receiving signals or other inputs from theprocessor(s) 110 and/or the autonomous driving module(s) 160. Anysuitable actuator can be used. For instance, the one or more actuators150 can include motors, pneumatic actuators, hydraulic pistons, relays,solenoids, and/or piezoelectric actuators, just to name a fewpossibilities.

The vehicle 100 can include one or more modules, at least some of whichare described herein. The modules can be implemented ascomputer-readable program code that, when executed by a processor 110,implement one or more of the various processes described herein. One ormore of the modules can be a component of the processor(s) 110, or oneor more of the modules can be executed on and/or distributed among otherprocessing systems to which the processor(s) 110 is operably connected.The modules can include instructions (e.g., program logic) executable byone or more processor(s) 110. Alternatively, or in addition, one or moredata store 115 may contain such instructions. Generally, the termmodule, as used herein, includes routines, programs, objects,components, data structures, and so on that perform particular tasks orimplement particular data types. In further aspects, a memory generallystores the noted modules. The memory associated with a module may be abuffer or cache embedded within a processor, a RAM, a ROM, a flashmemory, or another suitable electronic storage medium. In still furtheraspects, a module as envisioned by the present disclosure is implementedas an application-specific integrated circuit (ASIC), a hardwarecomponent of a system on a chip (SoC), as a programmable logic array(PLA), or as another suitable hardware component that is embedded with adefined configuration set (e.g., instructions) for performing thedisclosed functions.

In one or more arrangements, one or more of the modules described hereincan include artificial or computational intelligence elements, e.g.,neural network, fuzzy logic or other machine learning algorithms.Further, in one or more arrangements, one or more of the modules can bedistributed among a plurality of the modules described herein. In one ormore arrangements, two or more of the modules described herein can becombined into a single module.

The vehicle 100 can include one or more autonomous driving modules 160.The autonomous driving module(s) 160 can be configured to receive datafrom the sensor system 120 and/or any other type of system capable ofcapturing information relating to the vehicle 100 and/or the externalenvironment of the vehicle 100. In one or more arrangements, theautonomous driving module(s) 160 can use such data to generate one ormore driving scene models. The autonomous driving module(s) 160 candetermine position and velocity of the vehicle 100. The autonomousdriving module(s) 160 can determine the location of obstacles, or otherenvironmental features including traffic signs, trees, shrubs,neighboring vehicles, pedestrians, etc.

The autonomous driving module(s) 160 can be configured to receive,and/or determine location information for obstacles within the externalenvironment of the vehicle 100 for use by the processor(s) 110 , and/orone or more of the modules described herein to estimate position andorientation of the vehicle 100, vehicle position in global coordinatesbased on signals from a plurality of satellites, or any other dataand/or signals that could be used to determine the current state of thevehicle 100 or determine the position of the vehicle 100 with respect toits environment for use in either creating a map or determining theposition of the vehicle 100 in respect to map data.

The autonomous driving module(s) 160 either independently or incombination with the trajectory prediction system 170 can be configuredto determine travel path(s), current autonomous driving maneuvers forthe vehicle 100, future autonomous driving maneuvers and/ormodifications to current autonomous driving maneuvers based on dataacquired by the sensor system 120, driving scene models, and/or datafrom any other suitable source. “Driving maneuver” means one or moreactions that affect the movement of a vehicle. Examples of drivingmaneuvers include: accelerating, decelerating, braking, turning, movingin a lateral direction of the vehicle 100, changing travel lanes,merging into a travel lane, and/or reversing, just to name a fewpossibilities. The autonomous driving module(s) 160 can be configured toimplement determined driving maneuvers. The autonomous driving module(s)160 can cause, directly or indirectly, such autonomous driving maneuversto be implemented. As used herein, “cause” or “causing” means to make,command, instruct, and/or enable an event or action to occur or at leastbe in a state where such event or action may occur, either in a director indirect manner. The autonomous driving module(s) 160 can beconfigured to execute various vehicle functions and/or to transmit datato, receive data from, interact with, and/or control the vehicle 100 orone or more systems thereof (e.g. one or more of vehicle systems 140).

Detailed aspects are disclosed herein. However, it is to be understoodthat the disclosed aspects are intended only as examples. Therefore,specific structural and functional details disclosed herein are not tobe interpreted as limiting, but merely as a basis for the claims and asa representative basis for teaching one skilled in the art to variouslyemploy the aspects herein in virtually any appropriately detailedstructure. Further, the terms and phrases used herein are not intendedto be limiting but rather to provide an understandable description ofpossible implementations. Various aspects are shown in the collectivefigures, but the aspects are not limited to the illustrated structure orapplication.

The flowcharts and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousaspects. In this regard, each block in the flowcharts or block diagramsmay represent a module, segment, or portion of code, which comprises oneor more executable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved.

The systems, components and/or processes described above can be realizedin hardware or a combination of hardware and software and can berealized in a centralized fashion in one processing system or in adistributed fashion where different elements are spread across severalinterconnected processing systems. Any kind of processing system oranother apparatus adapted for carrying out the methods described hereinis suited. A typical combination of hardware and software can be aprocessing system with computer-usable program code that, when beingloaded and executed, controls the processing system such that it carriesout the methods described herein. The systems, components and/orprocesses also can be embedded in a computer-readable storage, such as acomputer program product or other data programs storage device, readableby a machine, tangibly embodying a program of instructions executable bythe machine to perform methods and processes described herein. Theseelements also can be embedded in an application product which comprisesall the features enabling the implementation of the methods describedherein and, which when loaded in a processing system, is able to carryout these methods.

Furthermore, arrangements described herein may take the form of acomputer program product embodied in one or more computer-readable mediahaving computer-readable program code embodied, e.g., stored, thereon.Any combination of one or more computer-readable media may be utilized.The computer-readable medium may be a computer-readable signal medium ora computer-readable storage medium. The phrase “computer-readablestorage medium” means a non-transitory storage medium. Acomputer-readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer-readable storage medium would include the following: a portablecomputer diskette, a hard disk drive (HDD), a solid-state drive (SSD), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a portable compact disc read-only memory (CD-ROM), adigital versatile disc (DVD), an optical storage device, a magneticstorage device, or any suitable combination of the foregoing. In thecontext of this document, a computer-readable storage medium may be anytangible medium that can contain or store a program for use by or inconnection with an instruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber, cable, RF, etc., or any suitable combination ofthe foregoing. Computer program code for carrying out operations foraspects of the present arrangements may be written in any combination ofone or more programming languages, including an object-orientedprogramming language such as Java™, Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer, or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

The foregoing description is provided for purposes of illustration anddescription and is in no way intended to limit the disclosure, itsapplication, or uses. It is not intended to be exhaustive or to limitthe disclosure. Individual elements or features of a particular aspectare generally not limited to that particular aspect, but, whereapplicable, are interchangeable and can be used in a selected aspect,even if not specifically shown or described. The same may also be variedin many ways. Such variations should not be regarded as a departure fromthe disclosure, and all such modifications are intended to be includedwithin the scope of the disclosure.

As used herein, the phrase at least one of A, B, and C should beconstrued to mean a logical (A or B or C), using a non-exclusive logical“or.” It should be understood that the various steps within a method maybe executed in different order without altering the principles of thepresent disclosure. Disclosure of ranges includes disclosure of allranges and subdivided ranges within the entire range, including theendpoints.

The headings (such as “Background” and “Summary”) and sub-headings usedherein are intended only for general organization of topics within thepresent disclosure, and are not intended to limit the disclosure of thetechnology or any aspect thereof. The recitation of multiple aspectshaving stated features is not intended to exclude other aspects havingadditional features, or other aspects incorporating differentcombinations of the stated features.

As used herein, the terms “comprise” and “include” and their variantsare intended to be non-limiting, such that recitation of items insuccession or a list is not to the exclusion of other like items thatmay also be useful in the devices and methods of this technology.Similarly, the terms “can” and “may” and their variants are intended tobe non-limiting, such that recitation that an aspect can or may comprisecertain elements or features does not exclude other aspects of thepresent technology that do not contain those elements or features. Theterms “a” and “an,” as used herein, are defined as one or more than one.The term “plurality,” as used herein, is defined as two or more thantwo. The term “another,” as used herein, is defined as at least a secondor more.

The broad teachings of the present disclosure can be implemented in avariety of forms. Therefore, while this disclosure includes particularexamples, the true scope of the disclosure should not be so limitedsince other modifications will become apparent to the skilledpractitioner upon a study of the specification and the following claims.Reference herein to one aspect, or various aspects means that aparticular feature, structure, or characteristic described in connectionwith an embodiment or particular system is included in at least oneembodiment or aspect. The appearances of the phrase “in one aspect” (orvariations thereof) are not necessarily referring to the same aspect orembodiment. It should be also understood that the various method stepsdiscussed herein do not have to be carried out in the same order asdepicted, and not each method step is required in each aspect orembodiment.

What is claimed is:
 1. A system for generating trajectories for avehicle user interface showing a driver's perspective view, the systemcomprising: one or more processors; and a memory communicably coupled tothe one or more processors and storing: a trajectory-prediction moduleincluding instructions that when executed by the one or more processorscause the one or more processors to: generate at least one road agentpredicted trajectory for a road agent that is external to anego-vehicle; determine that at least one road agent predicted trajectoryprotrudes from a display area and has an unknown direction of travelwhen displayed on the user interface showing a driver's perspectiveview; and modify the at least one road agent predicted trajectory toprovide an indication of direction; a control module includinginstructions that, when executed by the one or more processors, causethe one or more processors to update the user interface to include anymodified road agent predicted trajectory.
 2. The system according toclaim 1, wherein the instruction to provide an indication of directioncomprises an instruction to: modify a length of the at least road agentpredicted trajectory.
 3. The system according to claim 1, wherein theinstruction to provide an indication of direction comprises aninstruction to: add a directional arrow to an appropriate end of the atleast road agent predicted trajectory.
 4. The system according to claim1, wherein the instruction to provide an indication of directioncomprises an instruction to: add an icon representative of the roadagent to an appropriate end of the at least road agent predictedtrajectory to be visible when displayed on the user interface showing adriver's perspective view.
 5. A system for generating trajectories for avehicle user interface showing a driver's perspective view, the systemcomprising: one or more processors; and a memory communicably coupled tothe one or more processors and storing: a trajectory-prediction moduleincluding instructions that when executed by the one or more processorscause the one or more processors to: generate an ego-vehicle predictedtrajectory for an ego-vehicle; generate at least one road agentpredicted trajectory for a road agent that is external to theego-vehicle; determine that at least one predicted trajectory overlapseither an object or another predicted trajectory when displayed on theuser interface showing a driver's perspective view; and modify the atleast one predicted trajectory; and a control module includinginstructions that, when executed by the one or more processors, causethe one or more processors to update the user interface to include anymodified road agent predicted trajectory.
 6. The system according toclaim 5, wherein the trajectory-prediction module includes aninstruction to: determine that the at least one predicted trajectoryoverlaps one or more static object that does not reside on a road orsidewalk; and the instruction to modify the at least one predictedtrajectory comprises an instruction to dilute or blend the predictedtrajectory at each location where the predicted trajectory overlaps oneof the static objects.
 7. The system according to claim 5, wherein thetrajectory-prediction module includes an instruction to: determine thatthe at least one predicted trajectory is a past tense road agentpredicted trajectory; and the instruction to modify the at least onepredicted trajectory comprises an instruction to change one or both of ashape and thickness of the past tense road agent predicted trajectory.8. The system according to claim 5, wherein the instruction to modifythe at least one predicted trajectory comprises an instruction to changeone or both of a shape and dimension of the at least one predictedtrajectory.
 9. The system according to claim 8, wherein the shape anddimension of the at least one predicted trajectory is changed to bedifferent from a shape and dimension of the ego-vehicle predictedtrajectory.
 10. The system according to claim 5, wherein thetrajectory-prediction module further includes an instruction todetermine that a display in the user interface showing a driver'sperspective view is complex; and the control module further includes aninstruction to generate a display in the user interface showing a topplan view.
 11. The system according to claim 10, wherein the controlmodule further includes an instruction to obtain a selection requestfrom a user to display one of a top plan view and a driver's perspectiveview.
 12. The system according to claim 10, wherein the control modulefurther includes an instruction to generate a display in the userinterface showing both a top plan view and a driver's perspective view.13. A method for generating trajectories for a vehicle user interfaceshowing a driver's perspective view, the method comprising: generatingan ego-vehicle predicted trajectory for an ego-vehicle; generating atleast one road agent predicted trajectory for a road agent that isexternal to the ego-vehicle; determining that at least one predictedtrajectory overlaps either an object or another predicted trajectorywhen displayed on the user interface showing a driver's perspectiveview; modifying the at least one predicted trajectory; and updating theuser interface to include any modified road agent predicted trajectory.14. The method according to claim 13, further comprising determiningthat the at least one predicted trajectory overlaps a plurality ofstatic objects that do not reside on a road or sidewalk; and the step ofmodifying the at least one predicted trajectory comprises diluting orblending the predicted trajectory at each location where the predictedtrajectory overlaps one of the plurality of static objects.
 15. Themethod according to claim 13, further comprising: determining that theat least one predicted trajectory is a past tense road agent predictedtrajectory; and the instruction to modify the at least one predictedtrajectory comprises an instruction to change one or both of a shape andthickness of the past tense road agent predicted trajectory.
 16. Themethod according to claim 13, wherein the step of modifying the at leastone predicted trajectory comprises changing one or both of a shape anddimension of the at least one predicted trajectory.
 17. The methodaccording to claim 16, comprising changing the shape and dimension ofthe at least one predicted trajectory to be different from a shape anddimension of the ego-vehicle predicted trajectory.
 18. The methodaccording to claim 13, further comprising determining that a display inthe user interface showing a driver's perspective view is complex; andgenerating a display in the user interface showing a top plan view. 19.The method according to claim 18, further comprising obtaining aselection request from a user to display one of a top plan view and adriver's perspective view.
 20. The method according to claim 18, furthercomprising generating a display in the user interface showing both a topplan view and a driver's perspective view.